Focused Crawling: A Means to Acquire Biological Data from the Web

Experience paper. World Wide Web contains billions of publicly available documents (pages) and it grows and changes rapidly. Web search engines, such as Google and Altavista, provide access to indexable Web documents. An important part of a search engine is a Web crawler whose function is to collect Web pages for the search engine. Due to the Web’s immense size and dynamic nature no crawler is able to cover the entire Web and to keep up with all the changes. This fact has pushed the development of focused crawlers. In contrast to crawlers used by the general search engines focused crawlers selectively download Web documents, restricting the scope of crawling to a predefined domain. The downloaded documents can be stored and used as a source for data mining. In this paper we describe the main features of focused crawling, discuss the research on focused crawling conducted by the research group of the author, and discuss the problem areas associated with focused crawling not discussed in the literature which our work so far has revealed. Our test data consisted of Web documents in the genomics domain.